Repository navigation
(fix): Reject conflicting DRA kubelet-plugin healthcheck ports when ComputeDomains are enabled - #3029
jramkissoon wants to merge 2 commits into
Conversation
📝 WalkthroughWalkthroughThe DRA driver defines metrics ports for the GPU and compute-domains containers. It checks resolved healthcheck ports against enabled listener ports and returns a wrapped configuration error when positive ports conflict. Manifest generation performs this check before rendering. Tests cover collisions, disabled services, and non-colliding configurations. Priority: ⚪ Not assessed Merge Risk: 🟡 Moderate · up to Configurations with custom listener environment values can still render a DaemonSet whose listeners conflict. Reject those overrides or validate their effective values before merging, and add the missing collision tests.
Comment |
|
@jramkissoon are you still planning to work on this ? |
|
@kvalliyurnatt Yes, still on it. |
Review on NVIDIA#3029 pointed out that the kubelet-plugin skips its metrics listener when METRICS_PATH is empty, so a healthcheck on the freed port must be accepted. Prove that on the cluster: - compute-domains-metrics-disabled empties METRICS_PATH in both containers and moves their healthchecks onto 8080 / 8081. It must be accepted and both containers must stay ready, which also shows the user env entry wins over the template's METRICS_PATH=/metrics. - gpus-metrics-port-collision puts the gpus healthcheck on 8080 with metrics on. It must still be rejected with the DaemonSet untouched. scripts/metrics-path-proof.sh runs both plus recovery (make metrics-path-proof). post-fix-images.sh takes TAG so a before and an after build from a dirty tree get distinct tags and the operator rolls. Issue NVIDIA#2979 Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
Devin review on NVIDIA#3029: rejecting HTTP_ENDPOINT in kubeletPlugin.env breaks GPUClusters that moved the metrics listener, a configuration that works on released operators. compute-domains-http-endpoint-override moves the gpus metrics endpoint to :9090. scripts/http-endpoint-check.sh (make http-endpoint-check) deploys it and reports whether the operator accepts it, with metrics served on 9090 and not on 8080, or rejects it. EXPECT selects the passing outcome. v26.7.1: accepted, GET :9090/metrics 200, :8080 closed issue/NVIDIA#2979 head: rejected, "must not set HTTP_ENDPOINT" Issue NVIDIA#2979 Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
…t bed Review comment by kvalliyurnatt on PR NVIDIA#3029, internal/state/dra_driver.go line 152 (NVIDIA#3029 (comment)): "This is only checking for ports, if the ip addresses are different but port is same, this would falsely reject a valid configuration." validateKubeletPluginPorts keeps only the port of each HTTP_ENDPOINT, so two metrics listeners that share a port but bind different addresses are reported as a collision although both binds succeed inside the pod. Two overlays and one check script reproduce the condition on whichever operator is deployed: - compute-domains-metrics-distinct-hosts: gpus HTTP_ENDPOINT 127.0.0.1:9090, compute-domains HTTP_ENDPOINT 127.0.0.2:9090. The comment's case. - gpus-metrics-host-vs-healthcheck: gpus HTTP_ENDPOINT 127.0.0.1:51515 against the compute-domains health service on its default :51515. The counter-case: the health services bind the wildcard address, and Linux refuses a specific-address bind on a port with a wildcard listener. - scripts/distinct-hosts-check.sh deploys an overlay and reports accepted, rejected or collided. Accepted requires the render, no restarts over two liveness periods, the metrics server start line of each container on its configured address, and HTTP 200 on every address, dialed from inside the gpus container with bash /dev/tcp since port-forward only reaches 127.0.0.1. Collided means the render went through but a container lost its bind and restarted. EXPECT selects the outcome that exits 0. Results on a kind cluster with mock NVML (proof/distinct-hosts-check-*): PR build 98f910e (3a8c81a + test bed), distinct hosts: rejected: "the gpus container metrics endpoint (HTTP_ENDPOINT "127.0.0.1:9090") and the compute-domains container metrics endpoint (HTTP_ENDPOINT "127.0.0.2:9090") must differ; both resolve to 9090". DaemonSet left on :8080/:8081. Release v26.7.1, distinct hosts: accepted: both containers ready with 0 restarts over 65s, GET /metrics 200 on 127.0.0.1:9090 and on 127.0.0.2:9090, each container logging "Starting metrics HTTP server" on its own endpoint. PR build, gpus metrics 127.0.0.1:51515: rejected: "both resolve to 51515". Release v26.7.1, gpus metrics 127.0.0.1:51515: collided: compute-domains crash-looped with "failed to listen on :51515: listen tcp :51515: bind: address already in use" while gpus held the specific address. This confirms the comment for metrics-against-metrics: a port-only comparison rejects a configuration the kernel accepts. It also bounds the fix: a wildcard listener on either side still collides with every address on that port, so a host-aware validator must keep rejecting the second overlay. The health services always bind the wildcard address. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…est bed Review comment by kvalliyurnatt on PR NVIDIA#3029, internal/state/dra_driver.go line 94 (NVIDIA#3029 (comment)): "With METRICS_PORT=9090 followed by HTTP_ENDPOINT=:$(METRICS_PORT), Kubernetes supplies the plugin with :9090. The validator instead parses the literal expression and rejects it as nonnumeric, breaking previously valid configurations. I don't think just checking for nonnumeric is the right way to fix this." The kubelet expands $(NAME) in an env value against variables defined earlier in the same container. The template renders its own entries, HEALTHCHECK_PORT among them, before kubeletPlugin.env, and the user's entries follow in order, so both kinds of reference resolve. Two overlays reproduce the condition on whichever operator is deployed: - gpus-http-endpoint-expansion: METRICS_PORT=9090 then HTTP_ENDPOINT=:$(METRICS_PORT). The comment's case. - gpus-http-endpoint-expansion-collision: HTTP_ENDPOINT=:$(HEALTHCHECK_PORT), which expands to the gpus health port, 51516. A real collision that only an expansion-aware validator can reject for the right reason. scripts/distinct-hosts-check.sh becomes scripts/endpoint-check.sh, since it now covers both HTTP_ENDPOINT review comments. It distinguishes the rendered env value from the address the plugin binds after expansion, prints the rejection reason, and in the accepted path prints the plugin's startup flag dump, which shows the value the kubelet handed over. The distinct-hosts-check target keeps working; env-expansion-check is new. Results on a kind cluster with mock NVML (proof/endpoint-check-*): PR build a7f9ec1, :$(METRICS_PORT): rejected: HTTP_ENDPOINT ":$(METRICS_PORT)" must use a numeric port. PR build a7f9ec1, :$(HEALTHCHECK_PORT): rejected: HTTP_ENDPOINT ":$(HEALTHCHECK_PORT)" must use a numeric port. The right outcome for the wrong reason. Release v26.7.1, :$(METRICS_PORT): accepted: the DaemonSet carries the literal, the plugin's flag dump shows httpEndpoint ":9090", the metrics server starts on :9090 and GET /metrics answers 200; both containers ready with 0 restarts. Release v26.7.1, :$(HEALTHCHECK_PORT): collided: the gpus metrics server took 51516 and the health service crash-looped with "failed to listen for healthcheck service at :51516: bind: address already in use". This confirms the comment: a configuration the kubelet resolves into a working one is rejected for its literal text. The collision overlay shows why the fix should expand references the way the kubelet does rather than skip what it cannot parse: skipping would let a provable collision through. The operator image component points at the build of the previous commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review comment by kvalliyurnatt on PR NVIDIA#3029, internal/state/dra_driver_test.go line 485 (NVIDIA#3029 (comment)): "I think `:http` is valid and go resolves it to `:80`" The plugin passes HTTP_ENDPOINT straight to net.Listen, which accepts a service name for the port. The validator parses the port with strconv and rejects the name as nonnumeric; the test at line 485 asserts that rejection. Two overlays reproduce the condition on whichever operator is deployed: - gpus-http-endpoint-named-port: HTTP_ENDPOINT=:http. The comment's case. - gpus-http-endpoint-named-port-collision: HTTP_ENDPOINT=:http with the gpus healthcheck on 80, so both listeners of one container want the port. A real collision that only a validator resolving the name can reject for the right reason. endpoint-check.sh gains the two cases and a dial address separate from the plugin's: the plugin logs and reports ":http" as written, so the log and flag checks compare against that, while the in-pod HTTP check dials 80 numerically (bash's /dev/tcp needs a services table to resolve the name and the distroless image has none). named-port-check is new. Results on a kind cluster with mock NVML (proof/endpoint-check-*): PR build e451f72, :http: rejected: HTTP_ENDPOINT ":http" must use a numeric port. PR build e451f72, :http with healthcheck 80: rejected: HTTP_ENDPOINT ":http" must use a numeric port. The right outcome for the wrong reason. Release v26.7.1, :http: accepted: the DaemonSet carries ":http", the plugin's flag dump shows httpEndpoint ":http", the metrics server starts on ":http" and GET :80/metrics answers 200; both containers ready with 0 restarts. Release v26.7.1, :http with healthcheck 80: collided: the gpus metrics server took 80 and the health service crash-looped with "failed to listen for healthcheck service at :80: bind: address already in use". This confirms the comment: ":http" is a working listen address on released operators and the validator rejects it. The collision overlay shows the fix should resolve service names as net.Listen does, so a name that lands on a health port is still caught. The gpus container is privileged and on the pod network, so port 80 is bindable and nothing on the node competes for it. The operator image component points at the build of the previous commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…st bed Devin review on PR NVIDIA#3029, internal/state/dra_driver.go line 159 (NVIDIA#3029 (comment)): "When HTTP_ENDPOINT uses $(NODE_NAME) and a healthcheck port, metricsAddress skips the entire listener. On nodes whose name resolves locally, the health listener collides and the plugin crash-loops." The template sets NODE_NAME from a field reference ahead of kubeletPlugin.env, so the kubelet expands $(NODE_NAME) although the validator cannot know its value. The validator skips any endpoint that still holds a reference, which also discards a port it does know. One overlay reproduces the condition on whichever operator is deployed: - gpus-http-endpoint-node-host: HTTP_ENDPOINT=$(NODE_NAME):51516, the gpus health port. A metrics server on 51516 can never coexist with the health service on :51516, whatever the host resolves to. endpoint-check.sh gains the case. The bind-error search now reads the current log before the previous one, since in CrashLoopBackOff the latest failure is the current log, and the collided path prints the plugin's flag dump as evidence of the expansion. node-host-check is new; its default expectation is rejected, EXPECT=collided for an unfixed build. Result on a kind cluster with mock NVML (proof/endpoint-check-20261026T070727): Build 2f39ddb (all three earlier fixes), $(NODE_NAME):51516: collided. The spec is accepted and rendered; the plugin's flag dump shows httpEndpoint "issue-2979-worker:51516"; the name resolves to the node's Docker address, outside the pod, and the gpus container crash-loops with "listen tcp 172.18.0.2:51516: bind: cannot assign requested address". In this cluster the host resolves to a non-local address, so the metrics bind fails outright; a host resolving locally would lose to the health service with "address already in use" instead. Either way the spec cannot run, and the port alone proves it: the fix should keep a known port when only the host is unresolved and compare it against wildcard listeners. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Pushed the fixes for the review: f19b7c5 (host-aware comparison), |
|
@jramkissoon thanks for addressing the review comments, by trying to solve the metrics endpoint issue I think this has gotten pretty complex quickly due to the edge cases that we have discovered along the way, I am wondering if we can keep this PR down to the original scope of the issue which I think is the first two commits that you have ? we can revisit the metrics endpoint issue in a separate PR. wdyt ? |
|
Hello @kvalliyurnatt I do agree with the scope-creep, as currently this PR contains a larger volume of changes than I assume was initially expected. Two questions:
|
I would just make changes to this PR to have only the first two commits, remove the rest. Then squash and sign the first two commits. Also rebase the branch in this PR with main. |
0eec9ff to
66a73d8
Compare
|
Done: the PR is now the single commit 66a73d8, the original validator Local test bed used for scoped development work: Please review and confirm if changes to resolve devin-ai's review are required. If not, then this is complete per scope requirements. |
|
@jramkissoon can you use a GPG key to sign your commit. |
kvalliyurnatt
left a comment
There was a problem hiding this comment.
Could you update the PR description as well.
When ComputeDomains are enabled, the kubelet-plugin DaemonSet runs two containers that each bind a gRPC health service in the shared pod network namespace. With both configured on one port the second bind fails with EADDRINUSE and that container crash-loops; which one loses is a startup race. Validate the ports in the DRA driver state before rendering. The check runs on the resolved ports, so setting only one field to the other container's default is caught too, and a disabled health service is skipped. The error names both fields and the port and surfaces through the existing getManifestObjects path as a ReconcileFailed condition, leaving the DaemonSet untouched. Fixes NVIDIA#2979 Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
66a73d8 to
32e2a59
Compare
|
@kvalliyurnatt Ready for re-review |
|
/ok to test 32e2a59 |
|
@kvalliyurnatt Thanks for the approval |
kubeletPlugin.env is appended after the HEALTHCHECK_PORT entry the template renders from healthcheck.port, and the last duplicate wins in the container. A user entry therefore moves the health service behind the back of the port validation: with ComputeDomains enabled, gpus.kubeletPlugin.env HEALTHCHECK_PORT=51515 passes the check yet both containers bind 51515. Even without a collision the override cannot work, since the probes keep the spec port. Reject HEALTHCHECK_PORT in kubeletPlugin.env for the gpus container, and for the compute-domains container when it is rendered, pointing the user at the healthcheck.port field. Fixes NVIDIA#2979 (cherry picked from commit 709e7d3) Signed-off-by: jramkissoon <julianramkissoon@icloud.com>
Description
Fixes #2979
With
draDriver.computeDomains.enabled: truethe kubelet-plugin DaemonSet runstwo containers,
gpusandcompute-domains, each binding a gRPC health servicein the shared pod network namespace. Configured on the same port, the second
bind fails with
EADDRINUSEand that container crash-loops; which one loses isa startup race, so the symptom is hard to trace back to the GPUCluster spec.
The DRA driver state now validates the two healthcheck ports before rendering.
The check runs on the resolved ports, so setting only one field to the other
container's default (51516 / 51515) is caught too, and a disabled health
service never collides. A collision is reported through the existing
getManifestObjectserror path: the GPUCluster gets aReconcileFailedcondition naming both fields and the port, and the DaemonSet is left untouched.
Validation of the containers' metrics endpoints (
HTTP_ENDPOINT), raisedduring review, is deferred to a follow-up PR.
Checklist
make lint)make validate-generated-assets)make validate-modules)Testing
Unit tests:
TestDRADriverHealthcheckPortCollisioncovers equal explicitports, either port set to the other container's default, distinct ports, the
shipped defaults, computeDomains disabled, and one or both health services
disabled. The error cases fail against the pre-fix validator.
Manual testing on a kind cluster with mock NVML (nvml-mock via the run-ai
fake-gpu-operator), so both kubelet-plugin containers start and reach their
health bind:
computeDomains.kubeletPlugin.healthcheck.port: 51516:both containers render on 51516 and
compute-domainscrash-loops withfailed to listen on :51516: listen tcp :51516: bind: address already in use.probes on them, both containers ready with no restarts across two liveness
periods, GPUCluster
ready; the colliding spec is rejected withReconcileFailed: ... must differ when computeDomains is enabled; both resolve to 51516while the DaemonSet and both containers stay unchanged; restoring distinct
ports returns the GPUCluster to
ready.Test bed and captured evidence (not part of this PR):
https://github.com/jramkissoon/nvidia-gpu-operator/tree/gpu-operator-issue-%232979-rebased-26.7.2/deploy/issue-test-bed